OpenML in Python¶
OpenML is an online collaboration platform for machine learning:
- Share/reuse machine learning datasets, algorithms, models, experiments
- Well documented/annotated datasets, uniform access
- APIs in Java, R, Python*,... to download/upload everything
- Better reproducibility of experiments, reuse of machine learning models
- Works well with machine learning libraries such as scikit-learn
- Large scale benchmarking, compare to state of the art
Authentication¶
- Create an OpenML account (free) on http://www.openml.org.
- After logging in, open your account page (avatar on the top right)
- Open 'Account Settings', then 'API authentication' to find your API key.
There are two ways to authenticate:
- Create a plain text file
~/.openml/configwith the line 'apikey=MYKEY', replacing MYKEY with your API key. - Run the code below, replacing 'MYKEY' with your API key.
In [67]:
# If you don't keep your API key in the config file, you can also specify it in code (but never share your key!)
# oml.config.apikey = MYKEY
Data sets¶
We can list, select, and download all OpenML datasets
List datasets¶
In [68]:
datalist = oml.datasets.list_datasets() # Returns a dict
datalist = pd.DataFrame.from_dict(datalist, orient='index') # Create a DataFrame
print("First 10 of %s active datasets..." % len(datalist))
datalist[:10][['did','name','NumberOfInstances',
'NumberOfFeatures','NumberOfClasses']]
Out[68]:
There are many properties that we can query
In [69]:
list(datalist)
datalist = datalist[['did','name','NumberOfInstances',
'NumberOfFeatures','NumberOfClasses']]
Out[69]:
and we can filter or sort on all of them
In [70]:
datalist[datalist.NumberOfInstances>10000
].sort(['NumberOfInstances'])[:20]
Out[70]:
or find specific ones
In [71]:
datalist.query('name == "MagicTelescope"')
Out[71]:
In [72]:
datalist.query('NumberOfClasses > 50')
Out[72]:
Download a specific dataset. This is done based on the dataset ID (called 'did').
In [73]:
dataset = oml.datasets.get_dataset(1120)
print("This is dataset '%s', the target feature is '%s'" %
(dataset.name, dataset.default_target_attribute))
print("URL: %s" % dataset.url)
print(dataset.description[:500])
Convert the data to a DataFrame for easier processing/plotting
In [74]:
X, y, attribute_names = dataset.get_data(
target=dataset.default_target_attribute,
return_attribute_names=True)
magic = pd.DataFrame(X, columns=attribute_names)
magic['class'] = y
print(magic[:10])
In [75]:
magic.plot(figsize=(20,10))
pd.DataFrame(y).plot(figsize=(20,1));
In [76]:
magics = magic.sample(n=1000)
_ = pd.scatter_matrix(magics.iloc[:100,:4], c=magics[:100]['class'], figsize=(10, 10),
marker='o', hist_kwds={'bins': 20},
alpha=.8, cmap='plasma')
Train models¶
Train a scikit-learn model on the data manually
In [77]:
from sklearn import ensemble
from sklearn import model_selection
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y)
clf = ensemble.RandomForestClassifier(n_estimators=100)
clf.fit(X_train, y_train)
pd.DataFrame(clf.predict(X)).plot(figsize=(20,1));
And evaluate
In [78]:
kfold = model_selection.StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
results = model_selection.cross_val_score(clf, X, y, cv=kfold)
print("Accuracy: %.3f%% (+- %.3f)" % (results.mean(), results.std()))
Note: You can also ask which features are categorical to do your own encoding
In [79]:
from sklearn import preprocessing
dataset = oml.datasets.get_dataset(10) # Lymph dataset
X, y, categorical = dataset.get_data(
target=dataset.default_target_attribute,
return_categorical_indicator=True)
print("Categorical features: %s" % categorical)
enc = preprocessing.OneHotEncoder(categorical_features=categorical)
X = enc.fit_transform(X)
clf.fit(X, y)
Out[79]:
Tasks¶
To run benchmarks consistently (also across studies and tools), OpenML offers Tasks, which include specific train-test splits and other information to define a scientific task. Tasks are typically created via the website by the dataset provider.
Listing tasks¶
In [80]:
task_list = oml.tasks.list_tasks(size=5000) # Get first 5000 tasks
mytasks = pd.DataFrame(task_list).transpose()
print("First 5 of %s tasks:" % len(mytasks))
print(mytasks.columns)
In [81]:
mytasks = mytasks[['tid','did','name','task_type','estimation_procedure','evaluation_measures']]
print(mytasks.head())
Search for the tasks you need
In [82]:
print(mytasks.query('name=="MagicTelescope"'))
Download tasks¶
In [83]:
task = oml.tasks.get_task(3954)
pprint(vars(task))
Runs: Train models on tasks¶
We can run (many) scikit-learn algorithms on (many) OpenML tasks.
In [84]:
task = oml.tasks.get_task(3954)
clf = ensemble.RandomForestClassifier(n_estimators=100)
flow = oml.flows.sklearn_to_flow(clf)
run = oml.runs.run_flow_on_task(task, flow)
run.model
Out[84]:
Share the run on the OpenML server
- Note: if the exact same experiment has already been done (probably by you), it will just return the run ID from the previous run
In [85]:
myrun = run.publish()
print("Uploaded to http://www.openml.org/r/" + str(myrun.run_id))
It also works with pipelines¶
In [86]:
from sklearn import pipeline, ensemble, preprocessing
from openml import tasks,runs, datasets
task = tasks.get_task(59)
pipe = pipeline.Pipeline(steps=[
('Imputer', preprocessing.Imputer(strategy='median')),
('OneHotEncoder', preprocessing.OneHotEncoder(sparse=False, handle_unknown='ignore')),
('Classifier', ensemble.RandomForestClassifier())
])
flow = oml.flows.sklearn_to_flow(pipe)
run = oml.runs.run_flow_on_task(task, flow)
myrun = run.publish()
print("Uploaded to http://www.openml.org/r/" + str(myrun.run_id))
All together¶
Train any model on any OpenML dataset and upload to OpenML in a few lines of code
In [87]:
from sklearn.linear_model import LogisticRegression
task = oml.tasks.get_task(3954)
clf = LogisticRegression()
flow = oml.flows.sklearn_to_flow(clf)
run = oml.runs.run_flow_on_task(task, flow)
run.model
myrun = run.publish()
print("Uploaded to http://www.openml.org/r/" + str(myrun.run_id))
Out[87]:
- You can (of course) evaluate models in a loop
- E.g. for easy benchmarking:
In [89]:
import openml as oml
from sklearn import neighbors
for task_id in [14951,10103,9945]:
task = oml.tasks.get_task(task_id)
data = oml.datasets.get_dataset(task.dataset_id)
clf = neighbors.KNeighborsClassifier(n_neighbors=10)
flow = oml.flows.sklearn_to_flow(clf)
run = oml.runs.run_flow_on_task(task, flow)
myrun = run.publish()
print("kNN on %s: http://www.openml.org/r/%d" % (data.name, myrun.run_id))
Download everyone else's results on the same dataset¶
Check whether other people built better models on the same task by downloading their evaluations (computed on the OpenML server) and comparing directly against them.
In [90]:
myruns = oml.runs.list_runs(task=[3954],size=10000) # This can take a while
scores = []
for id, _ in myruns.items():
run = oml.runs.get_run(id)
if str.startswith(run.flow_name, 'sklearn'):
scores.append({"flow":run.flow_name, "score":run.evaluations['predictive_accuracy']})
In [94]:
import seaborn as sns
fig, ax = plt.subplots(figsize=(8, 12))
sns.violinplot(x="score", y="flow", data=pd.DataFrame(scores), scale="width", palette="Set3", cut=0);
